Distributed computing systems are an increasingly important part of research, governmental, and enterprise computing systems. Among the advantages of such computing systems are their ability to handle a variety of different computing scenarios including large computational problems, high volume data processing situations, and high availability situations. Such distributed computing systems typically utilize one or more storage devices in support of the computing systems operations. The storage devices used by distributed computing systems can be organized and interconnected using a storage area network (SAN), which is typically a high-performance network whose primary purpose is to enable storage devices to communicate with computer systems and with each other.
These storage devices can be quite numerous and/or heterogeneous. In an effort to aggregate such storage devices and to make such storage devices more manageable and flexible, storage virtualization techniques can be used within SAN devices such as switches, routers, hubs, management computer systems, and so-called “smart” storage devices (e.g., disk arrays, data movers, and third-party copy devices). Storage virtualization techniques establish relationships between physical storage devices, e.g. disk drives, tape drives, optical drives, etc., and virtual or logical storage devices such as volumes, virtual disks, and virtual logical units (sometimes referred to as virtual LUNs). In so doing, virtualization techniques provide system-wide features, e.g., naming, sizing, and management, better suited to the entire computing system than those features dictated by the physical characteristics of storage devices. Additionally, virtualization techniques enable and/or enhance certain computing system operations such as clustering and data backup and restore.
FIG. 1 illustrates a simplified example of a distributed computing system 100. The members of the distributed computing system include Server A 110 and Server B 120. As members of distributed computing system 100, servers 110 and 120 are often referred to as “hosts” or “nodes.” Thus, servers 110 and 120 are typically individual computer systems having some or all of the software and hardware components well known to those having skill in the art. FIG. 6 (described later in this application) illustrates some of the features common to such computer systems. Another common feature of a distributed computer system is the ability of the nodes to exchange data. In the example of FIG. 1, servers 110 and 120 can exchange data over network 150, typically a local area network (LAN), e.g., an enterprise-wide intranet, or a wide area network (WAN) such as the Internet. Additionally, network 150 provides a communication path for various client computer systems 140 to communicate with servers 110 and 120. In addition to network 150, servers 110 and 120 can communicate directly with each other over a private network (not shown). Private networks are typically used to support high availability where nodes 110 and 120 are part of a cluster, and include redundancy such as two network paths. Such networks can be used by the nodes for cluster service message passing including, for example, the exchange of so-called “heart-beat” signals indicating that each node is currently available to the cluster and functioning properly.
Other elements of distributed computer system 100 include storage area network (SAN) 130, SAN devices including virtualization functionality (“virtualizers”) 132 and 134, and storage devices such as tape library 170 (typically including one or more tape drives), a group of disk drives 180 (i.e., “just a bunch of disks” or “JBOD”), and intelligent storage array 190. These devices are examples of the type of storage used in cluster 100. Other storage schemes include the use of shared direct-attached storage (DAS) over shared SCSI buses. SAN 130 can be implemented using a variety of different technologies including fibre channel arbitrated loop (FCAL), fibre channel switched fabric, IP networks (e.g., iSCSI), Infiniband, etc.
While SAN 130 can be implemented using a variety of different network topologies, e.g., point-to-point, arbitrated loop, and switched or “fabric”, SAN 130 is shown as a switched network. In a switched SAN, each storage device and each SAN client is connected to one or more SAN devices, typically switches or routers, which make and break momentary connections between pairs of devices that need to communicate with each other. The term fabric is typically used to refer to such SAN configurations, portions of such SANs, and/or a collection of one or more SAN switches and client devices, such as hosts and storage.
SAN switches 132 and 134 are illustrated as virtualizers because each are designed to present one or more virtual storage devices, e.g., volumes or VLUNs, 133 and 135. Applications and system software (e.g., operating systems, file systems volume managers, etc.,) on hosts 110 and 120 and applications executing on client computer systems 140 can initiate or request I/O operations against virtual storage devices presented by virtualizers 132 and 134. These I/O operations typically include read and write operations to logical or virtual devices such as volumes, virtual LUNs, and/or virtual disks designed to appear and operate as SCSI LUNs. Virtualizers 132 and 134 in turn prepare physical I/O operations directed at various attached physical storage devices.
To further enhance functionality in distributed computer system 100, hosts 110 and 120 can each utilize more than one I/O port on SAN devices via dynamic multipathing (DMP) device drivers as well as multiple host bus adaptors (HBAs) (not shown). The HBAs provide multiple hardware interfaces between the host bus and the storage network for redundancy and/or I/O load balancing. Similarly, one or more of the physical storage devices can be connected to SAN devices through multiple I/O paths. DMP functionality generally enables greater reliability and performance by using path failover and load balancing. Multipathing policies used by DMP drivers depend on the characteristics of the devices in use. For example, active/active policies permit several paths to be used concurrently for I/O operations, thereby providing greater I/O throughput by balancing the I/O load uniformly across multiple paths. In the event of a loss of one connection, a DMP driver automatically routes I/O operations over the other available connection(s). Active/passive connections allow I/O operations on a primary (active) path while a secondary (passive) path is used if the primary path fails.
While there are many advantages to providing storage virtualization within a SAN, high availability of virtualization in the SAN depends on access by virtualization devices to back-end storage and availability of host access to the virtualization devices. Current high availability solutions consider each of these problems separately. Moreover, there are numerous ways in which device or communication path failure can partition a SAN fabric into two or more subfabrics resulting in the loss of access from an application host to a switch or from a switch to back-end devices. Such fabric partitions not only limit the usability of a SAN, but they can potentially lead to data corruption.
Accordingly, it is desirable to have scalable, flexible, and robust techniques for handling and repairing SAN fabric partitions, particularly where the SAN fabric includes more than one SAN device providing storage virtualization.